Inferential, non-parametric statistics to assess quality of probabilistic forecast systems

Many statistical forecast systems are available to interested users. In order to be useful for decision-making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and their statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of `quality’. However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what ‘quality’ entails and how to measure it, leading to confusion and misinformation. Here we present a generic framework to quantify aspects of forecast quality using an inferential approach to calculate nominal significance levels (p-values) that can be obtained either by directly applying non-parametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte-Carlo methods (in the case of forecast skill scores). Once converted to p-values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. Our analysis demonstrates the importance of providing p-values rather than adopting some arbitrarily chosen significance levels such as p < 0.05 or p < 0.01, which is still common practice. This is illustrated by applying non-parametric tests (such as KW and KS) and skill scoring methods (LEPS and RPSS) to the 5-phase Southern Oscillation Index classification system using historical rainfall data from Australia, The Republic of South Africa and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. We found that non-parametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system or quality measure. Eventually such inferential evidence should be complimented by descriptive statistical methods in order to fully assist in operational risk management.

Climate information needs of Gascoyne-Murchison pastoralists: a representative study of the Western Australian grazing industry

The Gascoyne-Murchison region of Western Australia experiences an arid to semi-arid climate with a highly variable temporal and spatial rainfall distribution. The region has around 39.2 million hectares available for pastoral lease and supports predominantly catle and sheep grazing leases. In recent years a number of climate forecasting systems have been available offering rainfall probabilities with different lead times and a forecast period; however, the extent to which these systems are capable of fulfilling the requirements of the local pastoralists is still ambiguous. Issues can range from ensuring forecasts are issued with sufficient lead time to enable key planning or decisions to be revoked or altered, to ensuring forecast language is simple and clear, to negate possible misunderstandings in interpretation. A climate research project sought to provide an objective method to determine which available forecasting systems had the greatest forecasting skill at times of the year relevant to local property management. To aid this climate research project, the study reported here was undertaken with an overall objective of exploring local pastoralists' climate information needs. We also explored how well they understand common climate forecast terms such as 'mean', median' and 'probability', and how they interpret and apply forecast information to decisions. A stratified, proportional random sampling was used for the purpose of deriving the representative sample based on rainfall-enterprise combinations. In order to provide more time for decision-making than existing operational forecasts that are issued with zero lead time, pastoralists requested that forecasts be issued for May-July and January-March with lead times counting down from 4 to 0 months. We found forecasts of between 20 and 50 mm break-of-season or follow-up rainfall were likely to influence decisions. Eighty percent of pastoralists demonstrated in a test question that they had a poor technical understanding of how to interpret the standard wording of a probabilistic median rainfall forecast. this is worthy of further research to investigate whether inappropriate management decisions are being made because the forecasts are being misunderstood. We found more than half the respondents regularly access and use weather and climate forecasts or outlook information from a range of sources and almost three-quarters considered climate information or tools useful, with preferred methods for accessing this information by email, faxback service, internet and the Department of Agriculture Western Australia's Pastoral Memo. Despite differences in enterprise types and rainfall seasonality across the region we found seasonal climate forecasting needs were relatively consistent. It became clear that providing basic training and working with pastoralists to help them understand regional climatic drivers, climate terminology and jargon, and the best ways to apply the forecasts to enhance decision-making are important to improve their use of information. Consideration could also be given to engaging a range of producers to write the climate forecasts themselves in the language they use and understand, in consultation with the scientists who prepare the forecasts.

Formative evaluation to benchmark and improve climate-based decision support for graziers in Western Queelsland

Researchers developing climate-based forecasts, workshops, software tools and information to aid grazier decisions undertook an evaluation study to enhance planning and benchmark impact. One hundred graziers in Western Queensland were randomly selected from 7 shires and surveyed by mail and telephone (43 respondents) to explore levels of knowledge and use of climate information, practices and information needs. We found 36% of respondents apply the Southern Oscillation Index to property decisions but 92% were unaware El Niño Southern Oscillation’s predictive signal in the region is greater for pasture growth than rainfall, suggesting they may not recognise the potential of pasture growth forecasts. Almost 75% of graziers consider they are conservative or risk averse in their attitude to managing their enterprise. Mail respondents (n= 20) if given a 68%, on average, probability of exceeding median rainfall forecast may change a decision; almost two-thirds vary stocking rate based on forage available, last year’s pasture growth or the Southern Oscillation Index; the balance maintain a constant stocking rate strategy; 90% have access to a computer; 75% to the internet and 95% have a fax. This paper presents findings of the study and draws comparisons with a similar study of 174 irrigators in the Northern Murray-Darling Basin (Aust. J. Exp. Ag. 44, 247-257). New insights and information gained are helping the team better understand client needs and plan, design and extend tools and information tailored to grazier knowledge, practice, information needs and preferences. Results have also provided a benchmark against which to measure project impact and have influenced the team to make important changes to their project planning, activities and methods for transferring technology tailored to grazier preferences.

Rainfall and streamflow response to El Niño Southern Oscillation: a case study in a semiarid catchment, Australia.

This paper aims to compare the shift in frequency distribution and skill of seasonal climate forecasting of both streamflow and rainfall in eastern Australia based on the Southern Oscillation Index (SOI) Phase system. Recent advances in seasonal forecasting of climate variables have highlighted opportunities for improving decision making in natural resources management. Forecasting of rainfall probabilities for different regions in Australia is available, but the use of similar forecasts for water resource supply has not been developed. The use of streamflow forecasts may provide better information for decision-making in irrigation supply and flow management for improved ecological outcomes. To examine the relative efficacy of seasonal forecasting of streamflow and rainfall, the shift in probability distributions and the forecast skill were evaluated using the Wilcoxon rank-sum test and the linear error in probability space (LEPS) skill score, respectively, at three river gauging stations in the Border Rivers Catchment of the Murray-Darling Basin in eastern Australia. A comparison of rainfall and streamflow distributions confirms higher statistical significance in the shift of streamflow distribution than that in rainfall distribution. Moreover, streamflow distribution showed greater skill of forecasting with 0-3 month lead time, compared to rainfall distribution.

Annual forecasting of the Australian macadamia crop - integrating tree census data with statistical climate-adjustment models

To facilitate marketing and export, the Australian macadamia industry requires accurate crop forecasts. Each year, two levels of crop predictions are produced for this industry. The first is an overall longer-term forecast based on tree census data of growers in the Australian Macadamia Society (AMS). This data set currently accounts for around 70% of total production, and is supplemented by our best estimates of non-AMS orchards. Given these total tree numbers, average yields per tree are needed to complete the long-term forecasts. Yields from regional variety trials were initially used, but were found to be consistently higher than the average yields that growers were obtaining. Hence, a statistical model was developed using growers' historical yields, also taken from the AMS database. This model accounted for the effects of tree age, variety, year, region and tree spacing, and explained 65% of the total variation in the yield per tree data. The second level of crop prediction is an annual climate adjustment of these overall long-term estimates, taking into account the expected effects on production of the previous year's climate. This adjustment is based on relative historical yields, measured as the percentage deviance between expected and actual production. The dominant climatic variables are observed temperature, evaporation, solar radiation and modelled water stress. Initially, a number of alternate statistical models showed good agreement within the historical data, with jack-knife cross-validation R2 values of 96% or better. However, forecasts varied quite widely between these alternate models. Exploratory multivariate analyses and nearest-neighbour methods were used to investigate these differences. For 2001-2003, the overall forecasts were in the right direction (when compared with the long-term expected values), but were over-estimates. In 2004 the forecast was well under the observed production, and in 2005 the revised models produced a forecast within 5.1% of the actual production. Over the first five years of forecasting, the absolute deviance for the climate-adjustment models averaged 10.1%, just outside the targeted objective of 10%.

Probabilistic forecasts of the onset of the north Australian wet season

The amount and timing of early wet-season rainfall are important for the management of many agricultural industries in north Australia. With this in mind, a wet-season onset date is defined based on the accumulation of rainfall to a predefined threshold, starting from 1 September, for each square of a 1° gridded analysis of daily rainfall across the region. Consistent with earlier studies, the interannual variability of the onset dates is shown to be well related to the immediately preceding July-August Southern Oscillation index (SOI). Based on this relationship, a forecast method using logistic regression is developed to predict the probability that onset will occur later than the climatological mean date. This method is expanded to also predict the probabilities that onset will be later than any of a range of threshold dates around the climatological mean. When assessed using cross-validated hindcasts, the skill of the predictions exceeds that of climatological forecasts in the majority of locations in north Australia, especially in the Top End region, Cape York, and central Queensland. At times of strong anomalies in the July-August SOI, the forecasts are reliably emphatic. Furthermore, predictions using tropical Pacific sea surface temperatures (SSTs) as the predictor are also tested. While short-lead (July-August predictor) forecasts are more skillful using the SOI, long-lead (May-June predictor) forecasts are more skillful using Pacific SSTs, indicative of the longer-term memory present in the ocean.

Needs for applied climate education in agriculture

This paper reports on a purposive survey study which aimed to identify needs for the development, delivery and evaluation of applied climate education for targeted groups, to improve knowledge and skills to better manage under variable climatic conditions. The survey sample consisted of 80 producers and other industry stakeholders in Australia (including representatives from consulting, agricultural extension and agricultural education sectors), with a 58% response rate to the survey. The survey included an assessment of (i) knowledge levels of the Southern Oscillation Index and sea surface temperatures, and (ii) skill and ability in interpreting weather and climate parameters. Results showed that despite many of the respondents having more than 20 years experience in their industry, the only formal climate education or training undertaken by most was a 1-day workshop. Over 80% of the applied climate skills listed in the survey were regarded by respondents as essential or important, but only 42% of educators, 30% of consultants and 28% of producers rated themselves as competent in applying such skills. Essential skills were deemed as those that would enable respondents or their clients to be better prepared for the next extended wet or dry meteorological event, and improved capability in identifying and capitalising on key decision points from climate information and a seasonal climate outlook. The complex issue of forecast accuracy is a confounding obstacle for many in the application of climate information and forecasts in management. Addressing this problem by describing forecast 'limitations and skill' can help to overcome this problem. The survey also highlighted specific climatic tactical and strategic information collated from grazing, cropping and agribusiness enterprises, and showed the value of such information from a users perspective.

Defining and predicting the 'break of the season' for north-east Queensland grazing areas

For pasture growth in the semi-arid tropics of north-east Australia, where up to 80% of annual rainfall occurs between December and March, the timing and distribution of rainfall events is often more important than the total amount. In particular, the timing of the 'green break of the season' (GBOS) at the end of the dry season, when new pasture growth becomes available as forage and a live-weight gain is measured in cattle, affects several important management decisions that prevent overgrazing and pasture degradation. Currently, beef producers in the region use a GBOS rule based on rainfall (e. g. 40mm of rain over three days by 1 December) to define the event and make their management decisions. A survey of 16 beef producers in north-east Queensland shows three quarters of respondents use a rainfall amount that occurs in only half or less than half of all years at their location. In addition, only half the producers expect the GBOS to occur within two weeks of the median date calculated by the CSIRO plant growth days model GRIM. This result suggests that in the producer rules, either the rainfall quantity or the period of time over which the rain is expected, is unrealistic. Despite only 37% of beef producers indicating that they use a southern oscillation index (SOI) forecast in their decisions, cross validated LEPS (linear error in probability space) analyses showed both the average 3 month July-September SOI and the 2 month August-September SOI have significant forecast skill in predicting the probability of both the amount of wet season rainfall and the timing of the GBOS. The communication and implementation of a rigorous and realistic definition of the GBOS, and the likely impacts of anthropogenic climate change on the region are discussed in the context of the sustainable management of northern Australian rangelands.

Using a process-based model, ParopSys, to predict the impact of climate change on the eucalypt plantation pest Paropsis atomaria.

The eucalypt leaf beetle, Paropsis atomaria Olivier, is an increasingly important pest of eucalypt plantations in subtropical eastern Australia. A process-based model, ParopSys, was developed using DYMEXTM and was found to accurately predict the beetle populations. Climate change scenarios within the latest Australian climate model forecast range were run in ParopSys at three locations to predict changes in beetle performance. Relative population peaks of early generations did not change but shifted to earlier in the season. Temperature increases of 1.0 to 1.5 ºC or greater predicted an extra generation of adults at Gympie and Canberra, but not for Lowmead, where increased populations of late season adults were observed under all scenarios. Furthermore, an additional generation of late-larval stages was predicted at temperature increases of greater than 1.0 ºC at Lowmead. Management strategies to address these changes are discussed, as are requirements to improve the predictive capacity of the model.

ACIAR Feasibility of farmers managing climate related crop production risk in India

Validation of new Indian seasonal climate forecasting products. In the Indian state of Andhra Pradesh (AP) kharif crops are heavily dependent on summer monsoon rains, where the timing and intensity of the rains affects crop yield. The majority of farms in AP are small and marginal, making them very vulnerable to yield reductions. Farmers also lack access to relevant information that might enable them to respond to seasonal conditions. Enabling farmers to utilise seasonal climate forecasting would allow them to respond to seasonal variability. To do this, farmers need a forecasting system that indicates a specific management strategy for the upcoming season, and effective and timely communication of the forecast information. Current agro-meteorological advisories in AP are issued on a bi-weekly basis, and they are relevant to an agro-climatic zone scale which may not be sufficiently relevant at a village level. Also, the information in the advisories may not be necessarily packaged in way relevant to cropping decisions by farmers. The objectives of this project are to evaluate the skill of seasonal climate forecasts to be issued for the 2008 monsoon season, to assess crop management options in response to seasonal scenarios that capture the range of seasonal climatic variability, to develop and evaluate options for effective communication and adoption of climate forecasts and agricultural advisories, and to synthesise and report on options for future research investments into seasonal climate forecasting.

Driving better vegetable irrigation through profitable practice change

The availability and quality of irrigation water has become an issue limiting productivity in many Australian vegetable regions. Production is also under competitive pressure from supply chain forces. Producers look to new technologies, including changing irrigation infrastructure, exploring new water sources, and more complex irrigation management, to survive these stresses. Often there is little objective information investigating which improvements could improve outcomes for vegetable producers, and external communities (e.g. meeting NRM targets). This has led to investment in inappropriate technologies, and costly repetition of errors, as business independently discover the worth of technologies by personal experience. In our project, we investigated technology improvements for vegetable irrigation. Through engagement with industry and other researchers, we identified technologies most applicable to growers, particularly those that addressed priority issues. We developed analytical tools for ‘what if’ scenario testing of technologies. We conducted nine detailed experiments in the Lockyer Valley and Riverina vegetable growing districts, as well as case studies on grower properties in southern Queensland. We investigated root zone monitoring tools (FullStop™ wetting front detectors and Soil Solution Extraction Tubes - SSET), drip system layout, fertigation equipment, and altering planting arrangements. Our project team developed and validated models for broccoli, sweet corn, green beans and lettuce, and spreadsheets for evaluating economic risks associated with new technologies. We presented project outcomes at over 100 extension events, including irrigation showcases, conferences, field days, farm walks and workshops. The FullStops™ were excellent for monitoring root zone conditions (EC, nitrate levels), and managing irrigation with poor quality water. They were easier to interpret than the SSET. The SSET were simpler to install, but required wet soil to be reliable. SSET were an option for monitoring deeper soil zones, unsuitable for FullStop™ installations. Because these root zone tools require expertise, and are labour intensive, we recommend they be used to address specific problems, or as a periodic auditing strategy, not for routine monitoring. In our research, we routinely found high residual N in horticultural soils, with subsequently little crop yield response to additional nitrogen fertiliser. With improved irrigation efficiency (and less leaching), it may be timely to re-examine nitrogen budgets and recommendations for vegetable crops. Where the drip irrigation tube was located close to the crop row (i.e. within 5-8 cm), management of irrigation was easier. It improved nitrogen uptake, water use efficiency, and reduced the risk of poor crop performance through moisture stress, particularly in the early crop establishment phases. Close proximity of the drip tube to the crop row gives the producer more options for managing salty water, and more flexibility in taking risks with forecast rain. In many vegetable crops, proximate drip systems may not be cost-effective. The next best alternative is to push crop rows closer to the drip tube (leading to an asymmetric row structure). The vegetable crop models are good at predicting crop phenology (development stages, time to harvest), input use (water, fertiliser), environmental impacts (nutrient, salt movement) and total yields. The two immediate applications for the models are understanding/predicting/manipulating harvest dates and nitrogen movements in vegetable cropping systems. From the economic tools, the major influences on accumulated profit are price and yield. In doing ‘what if’ analyses, it is very important to be as accurate as possible in ascertaining what the assumed yield and price ranges are. In most vegetable production systems, lowering the required inputs (e.g. irrigation requirement, fertiliser requirement) is unlikely to have a major influence on accumulated profit. However, if a resource is constraining (e.g. available irrigation water), it is usually most profitable to maximise return per unit of that resource.

Population dynamics of house mice in Queensland grain-growing areas

Context. Irregular plagues of house mice cause high production losses in grain crops in Australia. If plagues can be forecast through broad-scale monitoring or model-based prediction, then mice can be proactively controlled by poison baiting. Aims. To predict mouse plagues in grain crops in Queensland and assess the value of broad-scale monitoring. Methods. Regular trapping of mice at the same sites on the Darling Downs in southern Queensland has been undertaken since 1974. This provides an index of abundance over time that can be related to rainfall, crop yield, winter temperature and past mouse abundance. Other sites have been trapped over a shorter time period elsewhere on the Darling Downs and in central Queensland, allowing a comparison of mouse population dynamics and cross-validation of models predicting mouse abundance. Key results. On the regularly trapped 32-km transect on the Darling Downs, damaging mouse densities occur in 50% of years and a plague in 25% of years, with no detectable increase in mean monthly mouse abundance over the past 35 years. High mouse abundance on this transect is not consistently matched by high abundance in the broader area. Annual maximum mouse abundance in autumn–winter can be predicted (R2 = 57%) from spring mouse abundance and autumn–winter rainfall in the previous year. In central Queensland, mouse dynamics contrast with those on the Darling Downs and lack the distinct annual cycle, with peak abundance occurring in any month outside early spring.Onaverage, damaging mouse densities occur in 1 in 3 years and a plague occurs in 1 in 7 years. The dynamics of mouse populations on two transects ~70 km apart were rarely synchronous. Autumn–winter rainfall can indicate mouse abundance in some seasons (R2 = ~52%). Conclusion. Early warning of mouse plague formation in Queensland grain crops from regional models should trigger farm-based monitoring. This can be incorporated with rainfall into a simple model predicting future abundance that will determine any need for mouse control. Implications. A model-based warning of a possible mouse plague can highlight the need for local monitoring of mouse activity, which in turn could trigger poison baiting to prevent further mouse build-up.

Influence of different nitrogen rates and DMPP nitrification inhibitor on annual N2/O emissions from a subtropical wheat–maize cropping system

Global cereal production will need to increase by 50% to 70% to feed a world population of about 9 billion by 2050. This intensification is forecast to occur mostly in subtropical regions, where warm and humid conditions can promote high N2O losses from cropped soils. To secure high crop production without exacerbating N2O emissions, new nitrogen (N) fertiliser management strategies are necessary. This one-year study evaluated the efficacy of a nitrification inhibitor (3,4-dimethylpyrazole phosphate—DMPP) and different N fertiliser rates to reduce N2O emissions in a wheat–maize rotation in subtropical Australia. Annual N2O emissions were monitored using a fully automated greenhouse gas measuring system. Four treatments were fertilized with different rates of urea, including a control (40 kg-N ha−1 year−1), a conventional N fertiliser rate adjusted on estimated residual soil N (120 kg-N ha−1 year−1), a conventional N fertiliser rate (240 kg-N ha−1 year−1) and a conventional N fertiliser rate (240 kg-N ha−1 year−1) with nitrification inhibitor (DMPP) applied at top dressing. The maize season was by far the main contributor to annual N2O emissions due to the high soil moisture and temperature conditions, as well as the elevated N rates applied. Annual N2O emissions in the four treatments amounted to 0.49, 0.84, 2.02 and 0.74 kg N2O–N ha−1 year−1, respectively, and corresponded to emission factors of 0.29%, 0.39%, 0.69% and 0.16% of total N applied. Halving the annual conventional N fertiliser rate in the adjusted N treatment led to N2O emissions comparable to the DMPP treatment but extensively penalised maize yield. The application of DMPP produced a significant reduction in N2O emissions only in the maize season. The use of DMPP with urea at the conventional N rate reduced annual N2O emissions by more than 60% but did not affect crop yields. The results of this study indicate that: (i) future strategies aimed at securing subtropical cereal production without increasing N2O emissions should focus on the fertilisation of the summer crop; (ii) adjusting conventional N fertiliser rates on estimated residual soil N is an effective practice to reduce N2O emissions but can lead to substantial yield losses if the residual soil N is not assessed correctly; (iii) the application of DMPP is a feasible strategy to reduce annual N2O emissions from sub-tropical wheat–maize rotations. However, at the N rates tested in this study DMPP urea did not increase crop yields, making it impossible to recoup extra costs associated with this fertiliser. The findings of this study will support farmers and policy makers to define effective fertilisation strategies to reduce N2O emissions from subtropical cereal cropping systems while maintaining high crop productivity. More research is needed to assess the use of DMPP urea in terms of reducing conventional N fertiliser rates and subsequently enable a decrease of fertilisation costs and a further abatement of fertiliser-induced N2O emissions.

A participatory systems approach to understanding climate adaptation needs

Emerging literature on climate adaptation suggests the need for effective ways of engaging or activating communities and supporting community roles, coupled with whole-of-system approaches to understanding climate change and adaptation needs. We have developed and evaluated a participatory approach to elicit community and stakeholder understanding of climate change adaptation needs, and connect diverse community members and local office bearers towards potential action. The approach was trialed in a series of connected social-ecological systems along a transect from a rural area to the coast and islands of ecologically sensitive Moreton Bay in Queensland, Australia. We conducted ‘climate roundtables’ in each of three areas along the transect, then a fourth roundtable reviewed and extended the results to the region as a whole. Influence diagrams produced through the process show how each climate variable forecast to affect this region (heat, storm, flood, sea-level rise, fire, drought) affects the natural environment, infrastructure, economic and social behaviour patterns, and psychosocial responses, and how sets of people, species and ecosystems are affected, and act, differentially. The participatory process proved effective as a way of building local empathy, a local knowledge base and empowering participants to join towards future climate adaptation action. Key principles are highlighted to assist in adapting the process for use elsewhere.